
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2004-12 

Analyzing and sharing data for surface 
combat weapons systems 

Wilhelm, Gary L. 

Monterey, California. Naval Postgraduate School 


http://hdl.handle.net/10945/1299 


This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 

Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Caflwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional publicatkios created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. Caftiouo, NPS's first 
appointed — and published — schoteily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 Univefsity Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


THESIS 


ANALYZING AND SHARING DATA FOR SURFACE 
COMBAT WEAPONS SYSTEMS 

by 


Gary L. Wilhelm 


December 2004 


Thesis Advisor: 

Second Readers: 

Brad Naegle 

Erik Johnson 

Walter Owen 


Approved for public release; distribution is uniimited 




THIS PAGE INTENTIONALLY LEFT BLANK 



REPORT DOCUMENTATION PAGE 


FormA££roved^OMB^o^^0704^018^^^_ 
Public reporting burden for this collection of information is estimated to average 1 hour per response, ineluding the time 
for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing 
and reviewing the collection of information. Send oomments regarding this burden estimate or any other aspect of this 
collection of information, including suggestions for reducing this burden, to Washington headquarters Servioes, 
Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202- 
_4302^_and^ojTe_Officeof_Mana2ement_andBud2etj_Pa£erworl<^eduction_Pro|ectj6704^01^ 

I. AGENCY USE ONLY ^Leave Wanfc; I 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 

December 2004 Master’s Thesis 

4. TITLE AND SUBTITLE: Analyzing and Sharing Data for Surface 5. FUNDING NUMBERS 
Combat Weapons Systems 

6. AUTHOR(S) Gary L. Wilhelm _ 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION 

Naval Postgraduate School REPORT NUMBER 

9. SPONSORING /MONITORING AGENCY NAME(S) AND 10. SPONSORING/MONITORING 

ADDRESS(ES) AGENCY REPORT NUMBER 

N/A 

II. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not refleot the official 
policy or position of the Department of Defense or the U.S. Government. 

12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE 

Approved for public release; distribution is unlimited _ 

13. ABSTRACT (maximum 200 words) 

Test and evaluation of system performance has been a critical part of the acceptance of 
combat weapon systems for the Department of Defense. As combat weapon systems have 
become more complex, evaluation of system performance has relied more heavily on recorded 
test data. As part of the on-going transformation of the Defense department, Commercial-Off- 
The-Shelf (COTS) technology is being integrated into the acquisition of combat weapon systems. 
An Analysis Control Board (ACB) was created in response to these factors to support the AEGIS 
Weapon System Program Office. The focus of this ACB was to investigate and provide potential 
solutions to Data Dictionary, Data Recording and Data Reduction (R2D2) issues to the AEGIS 
Program Manager. This thesis discusses the history of the R2D2 ACB and its past, present and 
future directions. Additionally, this thesis examines how the R2D2 ACB concept could be applied 
to the DD(X) Next Generation Destroyer program. 


14. SUBJECT TERMS Analysis Control Board, Data Dictionary, Data Recording, Data 15. NUMBER OF 
Reduction, Test and Evaluation, AEGIS PAGES 

_93_ 

16. PRICE CODE 

17. SECURITY 18. SECURITY 19. SECURITY 20. LIMITATION OF 

CLASSIFICATION OF CLASSIFICATION OF THIS CLASSIFICATION OF ABSTRACT 

REPORT PAGE ABSTRACT 

Unclassified Unclassified Unclassified UL 

NSN 7540-01 -280-5500 Standard Form 298 (Rev. 2-89) 

Prescribed by ANSI Std. 239-18 


I 


























THIS PAGE INTENTIONALLY LEFT BLANK 



Approved for public release; distribution is unlimited 


ANALYZING AND SHARING DATA FOR SURFACE COMBAT WEAPONS 

SYSTEMS 

Gary L. Wilhelm 

Civilian, Naval Surface Warfare Center, Corona Division 
United States Navy 

B.S., University of California, Irvine, 1987 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN SYSTEMS ENGINEERING MANAGEMENT 


from the 


NAVAL POSTGRADUATE SCHOOL 
December 2004 


Author: Gary L. Wilhelm 


Approved by: Brad Naegle 

Thesis Advisor 

Erik O. Johnson 
Second Reader 

Walter E. Owen, DPA 
Second Reader 

Phil DePoy, Ph.D., Director, 

Wayne E. Meyer Institute of Systems Engineering 



THIS PAGE INTENTIONALLY LEFT BLANK 


IV 



ABSTRACT 


Test and evaluation of system performance has been a critical part of the 
acceptance of combat weapon systems for the Department of Defense. As 
combat weapon systems have become more complex, evaluation of system 
performance has relied more heavily on recorded test data. As part of the on¬ 
going transformation of the Defense department, Commercial-Off-The-Shelf 
(COTS) technology is being integrated into the acquisition of combat weapon 
systems. An Analysis Control Board (ACB) was created in response to these 
factors to support the AEGIS Weapon System Program Office. The focus of this 
ACB was to investigate and provide potential solutions to Data Dictionary, Data 
Recording and Data Reduction (R2D2) issues to the AEGIS Program Manager. 
This thesis discusses the history of the R2D2 ACB and its past, present and 
future directions. Additionally, this thesis examines how the R2D2 ACB concept 
could be applied to the DD(X) Next Generation Destroyer program. 
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EXECUTIVE SUMMARY 


Test and Evaluation is a critical part of the procurement of military systems 
and analysis of test data is required to determine if the system performance met 
the defined requirements. In an effort to ensure that the test data from formal 
acceptance tests, such as Ship Qualification Trials, Developmental and 
Operation Tests was adequate for analysis, an analysis control board (ACB) was 
formed to address Data Dictionary, Recording and Reduction (R2D2) issues. 
The R2D2 ACB has supported program managers in the AEGIS program office 
by investigating and providing technical expertise for Data Dictionary, Recording 
and Reduction issues. The R2D2 ACB is a proactive effort sponsored by the 
AEGIS program office to resolve issues that will affect the evaluation of formal 
acceptance testing before the tests are executed. Resolving issues prior to 
formal acceptance testing significantly reduces the risk of requiring a test to be 
re-executed. As the Defense Department transforms itself to meet future 
missions, this study examines Defense Transformation and SEA POWER 21 to 
determine how the R2D2 ACB can change to meet the challenges of the future. 

Changes in the design of military systems, specifically the implementation 
of Commercial-Off-The-Shelf hardware, have provided significant challenges to 
the recording and analysis of test data from the AEGIS Weapon System. As the 
AEGIS Weapon System completes the transition to Open Architecture, the 
challenge of assessing system performance will become even greater. This 
study includes an analysis of the DD(X) Next Generation Destroyer program and 
how the concept of the R2D2 ACB can be applied to this program. 
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I. INTRODUCTION 


A. BACKGROUND 

During the early development of the AEGIS computer program, data was 
recorded in a defined format called AEGIS Tactical Executive System for the 
AN/UYK-43 (ATES/43)'', or more commonly referred to as ATES. The function of 
data recording is to provide an objective set of information to evaluate system 
performance. Historically, the AEGIS program has recorded data in a binary 
format with the ATES format providing a “data recording function, which provides 
the means for extracting selected main memory resident data from tactical 
computer programs and for recording this data on media (optical disk or 
magnetic tape) for subsequent reduction and analysis.” (“Program Performance 
Specification”, 1999, p. 1-1) ATES was customized to allow the data to be 
captured from the unique, military standard hardware. This recording method 
focused on dumping data that was recorded in specific, defined, hard coded 
memory locations. Since the hardware was military standard (MIL-STD) and 
non-commercial, this method proved effective within the constraints of non¬ 
commercial hardware and software. The mission of ATES is defined “... as a 
subsystem within each of the combat system computer programs listed ... to 
provide for each of them an environment within which they can achieve their 
individual missions, and can work together to achieve the mission of the Combat 
System. This environment provides the ability for a program module to 
communicate with other program modules and data internal to the AN/UYK-43, 
the AN/UYK-43 hardware, and, via this hardware, the devices outside the 
AN/UYK-43.” (“Program Performance Specification”, 1999, p. 1-1) The data 
recording function was designed into the system during development providing 
an inherent advantage to later systems for event reconstruction but it was tied to 

"I The ATES data-recording format was designed to be used with MIL-SPEC computers by 
the AEGIS program. DXR was introduced with the insertion of COTS computers into the AEGIS 
Weapon System. ATES and DXR format the data received from the AWS and write the data in a 
binary format to a storage media, generally a digital tape or optical disk. When this data is 
extracted from the storage media, a data dictionary is used to reconstruct the data into the 
appropriate fields of information. 


1 



customized, MIL-STD hardware. ATES had some very specific capabilities 
embedded into the design of the data recording function including controlling 
application program module execution, fault tolerance to recover from all single 
point failures, support inter-computer communications and the data environment 
including data recording. (“Program Performance Specification”, 1999, p. 1-1) 

In the early development of the AEGIS Weapon System, data recording 
and extraction was a critical component of the development. The system 
designers and integrators relied upon the data extraction to determine system 
performance and specification compliance. However, data recording is not 
unique to the AEGIS program. Data recording and extraction is critical to test 
and evaluation events in order to determine whether the system performance 
was acceptable. Whether the data is recorded on a local disk drive or captured 
by telemetry from a satellite, the ability to reconstruct system performance is 
critical to determining is the mission was successfully accomplished. 

Baseline migration to Commercial-Off-the-Shelf (COTS) technology 
presented numerous data extraction/data reduction (DX/DR) challenges. A 
meeting was held at the Combat Systems Engineering Development Site 
(CSEDS), Moorestown, NJ on 25 July 2001 to discuss DX/DR issues associated 
with AEGIS Display System (ADS) Mark 6 and AEGIS Weapon System (AWS) 
Baseline 6 Phase 3. The discussion was focused on identifying current data 
extraction and reduction (DX/DR) short falls, assessing their impact on upcoming 
test events, and ensuring a solid plan was in place to correct discrepancies. Two 
major areas of ADS Mark 6 concerns were presented, 

1. Inability to perform complete event reconstruction at ADS and 

2. Data dictionary deficiencies limit the ability to reconstruct a 
complete set of console operator actions, operator alerts and 
doctrine. 

Cooperative Engagement Capability (CEC) provided the first at-sea 
developmental testing for the ADS Mark 6 aboard the USS HUE CITY and USS 
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VICKSBURG. The at-sea testing resulted in a large number of high priority 
issues. The lack of DX/DR capability for ADS Mark 6 was overshadowed as a 
result of the multitude of problems. The limited time and resources that were 
available at this time were devoted to developing the tactical baseline code that 
was viewed as the highest priority. As such, data analysis capability in ADS 
Mark 6 received a low priority during the two years of shipboard testing. While 
an ADS Baseline 6 Phase 1R data dictionary was provided during the later 
phases of Cooperative Engagement Capability (CEC) testing, enumerations to 
interpret numeric data were not provided, precluding meaningful analysis from 
being accomplished. Since there are no Extraction Points (EP) in ADS data 
dictionary for the track database corresponding the data displayed at the ADS 
consoles, complete reconstruction of tactical events as they were processed by 
key watch standers (e.g. Commanding Officer (CO), Tactical Action Officer 
(TAO)) at all 0-70 consoles was not possible. The lack of complete 
reconstruction of the tactical picture was a departure from the previous ATES 
baselines. The impact of this condition varies in severity according to the 
implementation of ADS in each particular baseline. Recent AEGIS baselines 
have captured some of this lost capability. 

Since the implementation of COTS technology for AWS was a phased 
approach, the analysis community realized that the problems faced with ADS 
were a precursor to similar problems that could result in the COTS 
implementation of the other elements of AWS. The length of production runs for 
COTS components is significantly shorter than the comparable MIL-SPEC 
components historically used in the AWS. The result of this was differing 
performance among equipment within the same AEGIS baseline. Additionally, 
the differences in COTS implementation between Cruisers and Destroyers 
resulted in multiple variants of the same AEGIS baseline. Different 
configurations within the Destroyers resulted in variants within the AEGIS 
Destroyers for Baseline 6 Phase 1. The spawning of multiple variants of the 
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AEGIS baselines resulted in large efforts to make the tactical computer programs 
ready for test and resulted in a resource limitation for non-tactical areas including 
data recording and extraction. 

While the problems with the ADS data recording provided the groundswell 
of concern that launched the Data Dictionary, Recording and Reduction Analysis 
Control Board (R2D2 ACB), the R2D2 ACB evaluates issues with recorded data 
for the SPY-1 radar, command and decision (C&D) and weapon control systems 
(WCS) as well. An analysis control board (ACB) is a technical working group that 
is chartered to deal specifically with data analysis issues. The ACB meetings are 
scheduled based upon the testing schedule and number and priority of 
outstanding issues. During the year 2003, the R2D2 ACB met approximately 
once every two months. Participants in the ACB included Naval Sea Systems 
Command, Naval Surface Warfare Centers (Corona, Dahlgren and Port 
Hueneme Divisions), Lockheed Martin, Northrop Grumman Mission Systems and 
others. 

The R2D2 ACB is part of the closed loop system engineering process that 
is used by the AEGIS program. Within this format, the R2D2 ACB reports to the 
program manager and is available to support the system engineering councils as 
required. The Closed Loop System Engineering process as implemented in the 
AEGIS Program is illustrated in Figure 1. 
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System Engineering Requirements 


• Operational Requirements 


• Previous Deficiencies 
• Added Capabilities 

• Previous Test Results 
■ SImulatlon/Valldatlon 



• Modify Requirements 
• Correct Training Deficiencies 

• Correct Performance Deficiencies 

■ Improve Capability 

• Determine Performance Limiters 

• Improve M&S Predictions 


• Test Objectives (TOs) 

■ Measures of Effectiveness 
• Test Kinematics 
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■ Performance Assessment 
■ Test Objective Assessment 
■ Daily Reports 


■ TO Assignment to Test Event 
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• Scenario Development 
• Performance Predictions 


• Observations 
■ Results 


• Test Plan 
Analysis Plan 
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i 


Plan Certification 


Test Execution 


Figure 1. Closed Loop System Engineering Process. 

(Sparks, 2001) 

In this closed loop system engineering (CLSE) process, system 
engineering requirements, operational requirements, deficiencies, new added 
capabilities, previous test results, and simulation/validation issues are evaluated. 
Test objectives are created to ensure that the following areas are properly 
analyzed. Measures of Effectiveness specify how each test objective is 
assessed and includes test kinematics, such as the required sequence of events, 
system settings, and target needs. Test objectives are assigned to specific test 
events. Test scenarios are developed to meet the objectives. Modeling and 
Simulation studies are performed to predict the expected outcome of each test 
scenario. Next, the requirements to execute the test scenarios are documented 
in the test plan. The test plan is required to complete a certification process that 
is completed about three months prior to the beginning of testing to allow 
sufficient time for test scenario preparations. During the CSSQT testing, the test 
scenarios are executed and the test data is collected and delivered to the 
analysis. Test objectives are assessed and performance issues become part of 
the Weapon System Performance Assessment (WSPA) process. Together, test 
objective assessment and the WSPA process feedback into the process. 
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resulting in modification of requirements, correction of training and performance 
deficiencies, capabilities improvements, determination of performance limiters, 
and improvement of M&S models. The test results are then fed into new test 
objectives and the closed loop system engineering process continues on. The 
R2D2 ACB is part of the assessment, reporting and system engineering 
requirements functions of this closed loop system engineering process. 

B. PURPOSE AND RESEARCH QUESTIONS 

The purpose of this study is to explore the lessons learned from the R2D2 
ACB and the value, if any, that it provides. This study will investigate the 
potential value that this type of forum can provided to other military weapon 
systems. The following research questions have been formulated to explore 
R2D2 ACB efforts and potential future directions. 


• Has the R2D2 ACB provided value to the AEGIS program? 

• What changes, if any, can be incorporated into the R2D2 ACB using best 
practices available today? 

• Is the Navy able to spend sufficient time supporting R2D2 issues? 

• What level of effort is required to successfully institute an R2D2 ACB and 

what level of support does the management/leadership need to provide for 
the ACB to provide value to each participant’s activity? 

• Can the R2D2 ACB concept be applied to the DDX program? 

• Can the R2D2 concept be applied to AEGIS Open Architecture? 


C. STAKEHOLDER INFORMATION 


There are many stakeholders in the R2D2 ACB. Each stakeholder is 
listed below with a brief description: 
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• Naval Sea Systems Command (NAVSEA) Program Executive Office 
Integrated Warfare Systems (PEO-IWS): Program office responsible for 
assignment of tasking. PEO-IWS 1A1B has oversight of the R2D2 ACB for 
the AEGIS program. 

• Naval Surface Warfare Center (NSWC), Corona Division: Tasked to be 
the independent assessment agent for NAVSEA/PEO-IWS 1 A. 

• Naval Surface Warfare Center, Dahlgren Division (NSWC DD): 

Responsible for AEGIS computer program certification. 

• Naval Surface Warfare Center, Port Hueneme Division: Performs the In- 
Service Engineering Agent (ISEA) function. 

• Computer Services Corporation (CSC): Responsible for developing 
computer code for the AEGIS program. 

• Lockheed Martin Maritime and Marine Systems (LM MMS): The prime 
contractor and design agent for the AEGIS program. 

• Combat Systems Engineering Development Site (CSEDS): Government 
owned facility where AEGIS computer program baselines are tested and 
functionality is demonstrated. 

• Northrop Grumman Mission Systems (NGMS): Contractor that supports 
the AEGIS program. NGMS has offices in Washington, DC to support the 
Program Office at WNY and in Mt. Laurel, NJ to support CSEDS. 


D. SURVEY QUESTIONS 


It is important to ensure that a representative of each stakeholder respond 
to the survey to ensure that the results reflect a consensus view. There are 
many methods that can be used and will be discussed in section 3. The survey 
is designed to provide insight into answering the research questions. 

E. SCOPE 


The scope of this study will examine three specific areas: AEGIS Baseline 
7 Phase 1, DD(X) platform and AEGIS Open Architecture (AOA). While the 
DD(X) platform is programmed to be an Open Architecture platform, it is unclear 
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as to the level of compatibility that the DD(X) and AOA implementations of Open 
Architecture will have. 

The R2D2 ACB has come into existence out of necessity. However, since 
its inception, no effort has been undertaken to quantify the benefits that it 
provides. This study will attempt to quantify the benefits that the R2D2 ACB 
provides to the AEGIS program. A high level review of efforts involved in 
supporting the ACB will also be characterized and compared to those benefits. 
After this comparison is completed, the DD(X) and Open Architecture programs 
will be examined to determine if the benefits provided to the AEGIS program will 
continue to be benefits to these emerging programs. A potential scope of costs 
will also be examined for comparison to the expected benefits. This study will 
also provide a roadmap for support of future AEGIS baselines that are all 
programmed to have some level of AOA implementation. 


F. BENEFITS OF STUDY 


There are three readily identifiable benefits of this study. The first benefit 
is that the R2D2 ACB will have a roadmap for supporting future programs. The 
second benefit is that the R2D2 ACB will be able to evaluate its present practices 
to determine if improvements can be put in place. The third and most important 
benefit is that this study will validate the value provided by the R2D2 ACB or if 
value is not provided, a recommendation to cease operations will be made. 
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II. LITERATURE REVIEW 


For this research study, a number of documents have been reviewed. 
Primary sources of information included R2D2 ACB presentations, meeting 
minutes and email messages. Additional sources included specifications, articles 
and selected publications. 


A. PROGRAM REVIEW 

1. R2D2ACB 


Data Dictionary, Recording and Reduction Analysis Control Board 
_ (R2D2 ACB) _ 


Co-Chair 


Co-Chair 
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uygs@navsea.navy.mil 

Gary Wilhelm PA34 
(909)273-4705 
gary.wilhelm@navy.mil 






NSWC Corona Lead 

NSWCDahlgren Lead 

NSWC Port Hueneme Lead 

Lockheed Martin Lead 

TecliRep Lead 


Michael Stetani 
(909) 273-5281 
michael.stefani@navy.mil 


Steven Canup 
(540) 653-6334 
CanupSE@nswc.navy.mil 


Jack Chung 
(805)228-8689 
chungjz@phdnswc.navy.mil 


Virginia Wall 
(856) 722-3167 
Virginia.a.wall@lmco.com 


LCDR Forster 


-► 

-► 

-► 

-► 

7.1 Lead 


-► 

-► 

-► 

7.1 Lead 


-► 

7.1 Lead 


-► 

-► 

-► 

-► 

-► 

7.1 Lead 


-► 

-► 

-► 

-► 

7.1 Lead 

Michael Slefani 
(909) 273-5281 
michael.stefani@navv.rail 

Paul Whichard 
(540) 653-1177 

(856) 722-2943 

LeeLK@pbdnswc.navy.mil 

Dave Griscora 
(856) 722-7474 

david.l.giiscom@lmco.com 

Mark Samara 
(856) 722-2042 

msamara@teclirep.navy. mil 

Data Dictionaries 

Data Dictionaries 


Data Dictionaries 

Data Dictionaries 

Data Dictionaries 

Stephen Cwinn 
(909) 273-1901 
StcDhen.Gwinn@navv.mil 

Steven Canup 
(540) 653-6334 

ranimSF.@n«wr navv rail 

Tao Doan 

Stacie Dahmen 
(856)914-7193 

Stacie.L.Dahmen@lmco.com 


EP Soec/ Pruurammini! 

EP Spec/ Prourammine 


EP SDec/Prourammini! 

EP Soec/Proerammine 
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Figure 2. R2D2 ACB Organizational Chart. 

(Gallagher, 2004). 

An ACB is a technical working group that relies upon participation from the 
entire AEGIS data analysis community, which ultimately answers to the AEGIS 


9 






















































































































program Office. The ACB strives to obtain group consensus for issues that must 
be resolved. While group consensus is not always possible, most issues are 
resolved with group consensus. In cases where a group consensus is not 
reached, a majority and minority viewpoint are presented to the program 
manager for ultimate adjudication if necessary. The R2D2 ACB has experienced 
development stages experienced by most working groups and passed through 
the stages of Forming, Storming, Norming, and Performing as depicted below. 
One of the challenges of this ACB has been to avoid “paralysis by analysis” as 
the group is composed of analysts. By encouraging discussion and respect of all 
viewpoints, the ACB has been able to extend the Performing stage of its 
development. (DAU Program Managers Tool Kit, 2003) 

WORKING GROUPS 


TEAM DEVELOPMENT WHEEL 


' X Performina 

/ Creative 

/ Trusting 

/ Effective 

1 Confident 

Forming 

Milling 

Confusion \ 

Polite \ 

Purposeless \ 

1 Norminq 

Storming / 

\ Cohesion 

Conflict J 

\ Purpose 

Fnjstration / 

Feedback 

Resistance / 

Relevancy 

Cliques^^^^^/ 


RECOGNIZE WHICH PHASE OF 
TEAM DEVELOPMENT YOU ARE 
IN AND TAKE POSITIVE ACTION 
TO WORK THROUGH 


Note: These can be an additional phase — ‘^Adjourning” — when the team 
disbands, says good bye^ and reflects on lessons learned. 

This is a “celebration” phase. 


Figure 3. Working Group Modei. 

(DAU Program Managers Tool Kit, 2003, p.79) 
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2. Coordinating Muiti-agency Testing and Satisfying Individuai 
Organizationai Requirements 

In the development of the AEGIS weapon system, each computer 
program baseline requires a series of tests. The tests required could include; 
functional configuration, demonstration, multi-element integration, computer 
certification, combat ship qualification trial, development and operational testing. 
Each of these tests may have a subset of stakeholders within the ACB and occur 
at different stages in the computer program development. One benefit of 
participation in the ACB has been the opportunity for involvement of all of the 
participants in earlier stages of computer program development. The charter of 
the ACB allows for a feedback mechanism to the computer program developer 
and integrator while keeping the involvement focused on issues that provide the 
greatest value. Early on, the prime contractor (LM) realized that receiving 
feedback on data recording issues earlier in the computer program development 
would help problems to be fixed before formal test and evaluation events. While 
having optimal data recording is desired, a dynamic balance exists that requires 
consideration of development of tactical functionality versus analysis capability. 
Capturing the value provided by the improvements in analysis capability allows 
the program manager to determine whether the benefit provided is worth the 
required resource expenditure. 

3. Selection of Team Members 

Analysis Team Member selection is effectively controlled by the program 
manager and exercises control by the budgeting of program funds. The 
chairman of the ACB has considerable influence in the selection and retention of 
team members as well. Each organization that is involved in the ACB relies upon 
the program manager to provide funding for many tasks beyond the ACB. The 
funding control allows the program manager to determine the level of 
participation required from each organization. If the participation is insufficient, 
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the program manager can require an activity to provide additional members to 
support the R2D2 ACB. Similarly, if a participant is not supporting the efforts of 
the ACB, the program manager can have that member removed from 
membership in the R2D2 ACB. 

4. Program Manager Support 

For the ACB to be successful, the program manager must support the 
ACB. Reasons for this include funding and acceptance of ACB findings. Without 
program manager support, the ACB would not be able to get an adequate level 
of participation to uncover and evaluate data dictionary, data recording and 
reduction issues. Even if the issues are discovered and documented, the 
program manager needs to understand and prioritize resources to address each 
issue relative to its risk to the program. 

5. Communication Methods 

Communication methods vary from informal to very formal. ACB meetings 
are generally communicated by a generic email sent to a large number of 
participants and interested parties. More formal communication has occurred at 
AEGIS Weapon System Performance Reviews (WSPR) where the R2D2 ACB 
presents major findings and issues to the AEGIS community. A website is used 
to post information and meeting minutes regarding the ACB to allow participant to 
review past events. 

6. Entrance Criteria 

The entrance criteria for consideration by the R2D2 ACB is that a 

computer program baseline is under development and is entering or in the Test 

and Evaluation phase. The exit criterion is correspondingly, when the Test and 
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Evaluation phase is complete including operational testing and the required 
analysis reports have been submitted. 


B. DETERMINATION OF STAKEHOLDERS 


The program manager and the ACB chairman determine stakeholders. 
Any organization or person desiring to join the ACB can make a request to the 
program manager or ACB chair. All direct participants in the ACB are considered 
to be direct or indirect stakeholders by virtue of their ACB status. 


C. ROLES AND RESPONSIBILITIES 


The roles of the ACB fall into a few categories as listed below: 


• Program Manager: Final authority on whether funding can be provided to 
remedy data dictionary, recording and reduction issues. Provides high- 
level interface to the program manager and executive management levels 
at the contractor and within the Program Executive Office. 

• ACB Chairman: Responsible for determining issues that merit discussion 
and setting agendas. Reports directly to the program manager and 
provides a single voice representing the analysis community consensus 
on issues discussed. The chairman provides an objective view that does 
not favor any individual stakeholder or member. 

• Leader: Each stakeholder activity designates a leader for their 
representation. The leader is expected to provide a consensus view for 
each issue for their activity. The leader is also responsible for ensuring 
that action items assigned to members from their activity are answered. 

• Participant: A participant is someone who regularly or periodically 
attends ACB meetings. Their level of involvement varies from significant 
to spectator depending on issues that require discussion and action. 
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D. 


CURRENT EVALUATION PLAN 


The AEGIS baseline managers evaluate the efforts of the R2D2 ACB and 
their support for ACB efforts is required for success. If the R2D2 ACB is not 
performing well, the baseline manager will not support the R2D2 ACB. Lack of 
participation by the baseline manager indicates to the contractor that the issues 
that the ACB surfaces are not important enough to warrant expenditure of 
significant resources. However, if the baseline manager is supportive and 
involved, the effort expended to remedy issues will increase as the contractor 
views the ACB as an extension of the program office. 


E. IDENTIFIED RISKS / CONCERNS 

1. Present Current Data Dictionary, Recording and Reduction 
Issues 

The R2D2 ACB meets approximately once every two months to discuss 
and evaluate issues regarding data dictionaries, data recording and data 
reduction. If an issue cannot be resolved during the meeting, an action item is 
assigned to collect the information required for the ACB to make an informed 
decision on the importance of the issue and the potential risk that it may cause 
the program. The action items are tracked by the ACB until closed and the 
status is reported to the program manager. The combination of meeting minutes 
and action items provide a comprehensive list of all data dictionary, recording 
and reduction issues addressed by the ACB. 

2. Team Effectiveness 

As part of this research effort, a review of team effectiveness practices 

was conducted. While the R2D2 ACB is a technical working group, at a practical 

level, it must be able to consider the policy implications of the issues to be 
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presented to the program manager. The selection of A Practical Guide for Policy 
Analysis by Eugene Bardach was chosen to provide some insight and tools for 
this reality. This portion of the literature review details the eightfold path problem 
solving method. (Bardach, 2000, p. 1-46) The method steps are: 


1 . Define the Problem. Defining the problem should be evaluative 
and quantifiable. Find conditions that cause problems. Look for 
any opportunities that are being missed. Make sure that to avoid 
fitting a predefined solution into the defined problem and be 
skeptical when cause and effect are presented. (Bardach, 2000, p. 
1 - 6 ) 

2. Assemble some Evidence. Think about the value of the evidence 
before spending resources to collect it. Conduct a literature review 
and survey best practices to determine if any apply to this problem. 
(Bardach, 2000, p. 7-12) 

3. Consider Alternatives. Model the system and stay focused on the 
problem as alternatives are evaluated. (Bardach, 2000, p. 12-19) 

4. Select Criteria. When selecting criteria, think about judging the 
outcomes and determine how to weigh aspects such as efficiency, 
acceptability, improvement and robustness. (Bardach, 2000, p. 19- 
26) 

5. Project the Outcomes. Projection = Model -i- Evidence. Estimate 
the magnitude of the outcome and consider where is the break¬ 
even point. Don’t be too optimistic and remember to consider the 
undesirable side effects. (Bardach, 2000, p. 27-36) 

6. Confront the Trade-offs. If a good job is done is step 5, there 
should be plenty of good choices of outcomes to try to optimize for 
a solution. (Bardach, 2000, p. 37-39) 

7. Decide. If the results of these steps indicate that the project is not 
worth doing, then go back and revisit steps 1 through 6. Just 
because it hasn’t been done before doesn’t mean it isn’t a good 
solution even if it seems too obvious. As Bardach wrote, “If your 
favorite policy alternative is such a great idea, how come it’s not 
happening already? The most common source of failure on this 
test is neglecting to consider the resistance of bureaucratic and 
other stakeholders in the status quo”. As changes are identified in 
a project, the bureaucratic and other stakeholders must be 
considered and how to achieve their buy-in to avoid and overcome 
resistance. (Bardach, 2000, p. 40-41) 
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8. Tell your story. Consciously avoid some of the common pitfalls 
which include: following the eightfold path without thinking it 
through, compulsive qualifying, show your work but stay focused on 
what is really important, listing without explaining will not help the 
readers and evaluators to understand your decisions, avoid a 
pompous insider’s style. (Bardach, 2000, p. 41-46) 

3. Project Management 

The R2D2 ACB requires careful consideration of the management of 
projects. Visualizing Project Management by K. Forsberg, H. Mooz, and H. 
Cotterman (2000) provides the following practical advice in project management. 
Eliminating features from a proven template must be justified and to proactively 
manage a project requires an approach that is “orderly, methodical and 
disciplined”. (K. Forsberg, FI. Mooz, and FI. Cotterman, 2000, p. 77) During 
Combat System Ship Qualification Trials (CSSQT), the analysis of weapon 
system performance requirements must be balanced with the computer program 
development resources and the test and evaluation schedule before any 
changes are requested to the test events. The ACB is cognizant about trying to 
gain efficiency at the expense of eliminating a necessary function. The ACB 
function of conducting analysis acts as a control gate with reporting to the 
program office and numerous System Engineering Councils (SEC). 

There are three aspects to the project cycle: business, budget, and 
technical. (K. Forsberg, FI. Mooz, and FI. Cotterman, 2000, p. 87-88) A template 
for a project cycle that one can compare to a list of events is given in the next 
figure. It is unwise to allow the pursuit of ‘better, cheaper, and faster’ to allow the 
discarding of controls and elimination of tasks without regard to the 
consequences. “The key to success is to design a tailored, gated cycle that is 
based on a proven template but that is lean, efficient, and effective.” (K. 
Forsberg, FI. Mooz, and FI. Cotterman, 2000, p. 107) 
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4. 


Types of Testing 


In the acquisition cycle for the AEGIS Weapon System, many different 
levels of testing are performed. Based upon the past R2D2 ACB efforts, the 
phases of testing that have been of particular importance to the R2D2 ACB are: 


1. Land Based Computer Program Demonstration Test - A formal 
test that is performed to demonstrate computer program 
functionality to demonstrate compliance to the top-level 
specification. 

2. System Program Certification Test - Combat system certification 
evaluates a computer programs ability to perform required ship 
missions in accordance with specifications, validates computer 
programs do not regress from the capability of the programs being 
replaced, verifies the computer programs are stable in use and that 
the computer programs can be safely operated under normal 
operating conditions. 

3. Combat System Ship Quaiification Triai - A set of at-sea test 
and evaluation events to demonstrate weapon system functionality 
and the ability to perform required ship missions in accordance with 
specifications on a specific surface ship platform. 

4. Developmental Test (DT) - Testing to determine technical 
performance and specification compliance. 

5. Operational Test (OT) - Testing to determine the operational 
effectiveness of the system 

Figure 4 is taken from the Program Managers Toolkit and provides a very 
succinct description of the Test and Evaluation phase. 
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Figure 4. Test and Evaluation 

(DAU Program Managers Tool Kit, 2003, p.51) 


5. Risk Management and Acquisition Planning 


As part of the literature review, some key points and concepts found in the 
Software Engineering Institute’s Team Risk Management Model and the 
Guidelines for Successful Acquisition and Management of Software-Intensive 
Systems (GSAM) are provided. Risk management is a critical part of every 
acquisition program. An effective risk management approach will identify and 
plan for areas of risk in a program and analyze those risks. Once this is 
accomplished, a plan to monitor and control the areas of risk can be put into 
place. To effectively manage risk in an optimal program, trade-offs between 
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cost, performance and schedule will need to be performed. Risk management 
has been defined as “a discipline and environment of proactive decisions and 
actions to 

1. assess continuously what can go wrong (risks). 

2. determine what risks are important to deal with. 

3. implement strategies to deal with those risk.” 

(Higuera, R, Dorofee, A, Walker, J, Williams, R, 1994, p. 6), 

At the center of effective risk management is communication. Without 
effective communication, the impact of an area of risk cannot be adequately 
assessed and dealt with. For risk management to be effective, it takes more than 
just the Program Manager. At all levels of the program and every phase from 
design to development to production, risks must be identified and mitigated 
where possible. When a risk cannot be effectively mitigated due to resource or 
other constraints, it should be documented to ensure that the program cost, 
schedule and performance requirements are effectively managed. (GSAM, 2000, 
p. 6-10) The intent is to integrate risk management into the program team efforts 
and not allow risks to proceed into a program undetected. As demonstrated in 
Figure 5, risk management requires trading off estimated cost versus potential 
losses. 
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Figure 5. Cost/Benefits of Effective Risk Management 

(GSAM, 2000, p.6-6) 
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Risk mitigation is an important part of risk management. In this process, 
risk is identified and evaluated. Once this is completed, options are considered 
to set the identified risks at an acceptable level within the resource constraints. A 
critical part of this process includes assessing the possible consequences of 
inaction. Ignoring predictable risks is a common problem that the program 
manager needs to be aware of. As each risk item proceeds through the risk 
management process, the decision on how to deal with each risk item must be 
communicated to the program manager. Without authorization from the PM, 
resources cannot be allocated and preventative action cannot be undertaken. 
One risk that is true for all acquisition programs is funding. Ensuring that “a well- 
defined set of requirements and active management involvement” can help to 
mitigate this area of risk. The levels of risk are listed in Table 1. 
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Magnitude 

Guidelines 

Critical 

High likelihood of severely impacting 

one or more factors, i.e., cost & 

schedule, performance, or 

supportability. 

High 

High likelihood of moderately impacting 

one or more factors. 

Medium 

Medium likelihood of moderately 

impacting one or more factors. 

Low 

Low likelihood of moderately impacting 

one or more factors. 


Table 1. Level of Risk Management and Guidelines 

(GSAM, 2000, p.6-41) 

Software development programs can possess many subtle environmental 
risk factors (GSAM, 2000, p.6-36). Software developments can be very complex. 
Integration of different modules can result in complex interactions that are not 
easily resolved. Problem elements have multidimensional relationships. Adding 
more people to a project may not increase effectiveness. In fact, it may reduce 
the productivity of the team. Software problems can change and are inherently 
unstable. Actual costs and time to develop are difficult to accurately project. 
Software development is dynamic. Due to changes in the requirements and 
resources, the development progress and environment will continuously change. 

Software development requires people who represent a major source of 
risk. Conflicts in human environment, interaction and desires will cause 
problems since software development is a “very human endeavor” (GSAM, 2000, 
p.6-36). Software development requires performing up-front, strategic planning 
to address problems that will be more costly to fix in the later phases. In fact, 
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“The time you spend defining your acquisition strategy early on will go a long way 
in assuring stability throughout the entire life of the system...” (GSAM, 2000, p.6- 
36) The use of tools such as work breakdown structures can assist in linking the 
strategic goals with the software development efforts. The use of open systems 
is expected to improve the supportability of software systems. Some classic 
mistakes in software acquisition include unrealistic estimates in resources 
required for development, inadequate software test and integration, significant 
code development before requirements are stable. 
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III. R2D2 ACB CASE STUDY 


A. CASE STUDY METHODS - PAST PERFORMANCE - WHAT HAS THE 

R2D2 DONE? 

The R2D2 ACB meetings are generally conducted using a Video 
Teleconference format. This medium has been selected to maximize 
stakeholder participation and interaction. Early on, it was determined that, due to 
the locations of the stakeholders, it would be difficult to get a consensus 
participation in one location. This ruled out the traditional announce a meeting 
and have everyone meet at one location, typically the Washington Navy Yard 
(WNY). Many of the issues that the ACB had to deal with have been complex. 
Discussing complex, data related issues; over a voice only teleconference was 
determined to be ineffective for understanding many of the issues. Email and 
electronic discussion boards were considered and were determined to be useful 
as tools, but not as the primary method of interaction. 

The use of video teleconferencing has maximized participation from all 
stakeholders. The video teleconference format allows participation from the 
activity site and is the least disruptive option for stakeholder participation 
compared to on site meetings at WNY. For a typical meeting, there are 3 to 4 
nodes. A node is a site that has a VTC connection. For example, the R2D2 ACB 
meeting in 29 April 2004 had 3 nodes. The nodes were at: 


• NGMS, Washington DC: 80 M Street SE, Suite 500 Chesapeake Room 
5123 

• NSWC Corona: Building 511, Room 109A 

• LM MMS, Moorestown, NJ: 4000 Building Room 374 

Email communication is essential to the effective operation of the R2D2 
ACB and meeting announcements are promulgated using this method. Email is 
frequently where issues are first socialized to determine if their importance and 
level of program impact merits discussion during a meeting. 
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Another tool that is used by the ACB is the Corporate Document 
Management System (CDMS). CDMS allows all documents releasable by the 
R2D2 ACB to be accessed by all participants. This has provided many benefits. 
The primary benefit is that large files such as presentations from meetings do not 
need to be emailed to all participants. Many participants are only interested in a 
subset of the issues discussed by the ACB and CDMS allows each participant to 
retrieve only in the information that interests them. An additional benefit is that 
participants’ mailboxes would not be filled up with large files from the ACB 
meetings or information distributions. Another benefit provided by CDMS is that 
it provides a historical record of information on topics discussed. When a new 
member joins the R2D2 ACB as a participant, CDMS can be accessed to find 
previous information on topics of interest, allowing a new member to learn quickly 
about topics of interest. 

The R2D2 ACB has assigned a total of 194 action items since its 
inception. Of these action items, only 18 action items remain open as of 15 
August 2004. 

B. SURVEY - FORMATS AND METHODOLOGY 


Various methods were considered for a survey to gather information to 
answer the research questions posed in this thesis. Options for the survey 
included (University of Phoenix, 2002, p. 271-280): 


Multiple Choice and True/False questions - Multiple choice and True or 
False questions were not used since they would imply a known set of 
answers. These types of responses are unlikely to provide insight into 
future directions. An objective of this research is to determine the 
likelihood of success for potential future opportunities and which directions 
are likely to be successful. 

Ad Hoc questions - Ad Hoc method where questions are improvised 
during the survey or posed only of certain survey participants were not 
selected. The analysis of the data from an Ad Hoc survey is problematic 
as similar questions may be asked but the differences are subtle enough 
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to preclude having the data correlated. Ad Hoc surveys can lead to small 
sample sizes as most questions are unique and results cannot be easily 
combined. 

Open-ended questions - Open-ended questions were not used for the 
survey. Open-ended questions do not provide a standardized set of 
responses that can be easily correlated. 

Closed-ended questions - Closed-ended questions were used for the 
survey. Since the survey was distributed by electronic format and follow¬ 
up was conducted in person or by telephone, standardization of responses 
was a critical consideration. 

Based upon discussions with the ACB Co-Chair, Geoff Uy, and ACB 
member Michelle Gallagher, the five questions were chosen for this survey. The 
questions were designed to provide insight into the success of past efforts and 
likelihood of success future opportunities for the R2D2 ACB. (University of 
Phoenix, 2002, p. 271-280): 


• How effective in providing value to the AEGIS program has the R2D2 ACB 
been? 

• How effective do you expect the R2D2 ACB to be in AEGIS Open 
Architecture baselines? 

• How would you characterize the amount of time you are allowed to support 
R2D2 issues? 

• At your activity, how well does the management/leadership understand and 
support your involvement in the R2D2 ACB? 

• How much benefit do you think the R2D2 ACB concept would provide to a 
new weapon system program? 


The following choices for possible answers to each question were 


provided as follows: 


1. Excellent or Extremely Effective 

2. Good or Very Effective 

3. Fair or Effective 

4. Poor or Ineffective 

5. No Opinion 
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The selection of these choices will allow for a quantitative and qualitative 
analysis. (University of Phoenix, 2002, p. 271-280) 


C. OPEN ARCHITECTURE 


Open architecture is a concept that is being put into practice that allows 
for information to flow across boundaries and allow for interoperability between 
systems or components. The OPEN Group defines the characteristics of 
information flow across system boundaries as: 

“It has open standard components that provide services in a customer's 
extended enterprise that: Combine multiple sources of information. Deliver 
information to the places where that information is needed, and In the right 
context for the people or systems using that information. “(Blevins, T., 2004, f24) 

Designing open systems is a challenge not only to the commercial 
industry but for military systems as well. The Open Systems Joint Task Force 
has defined Open Systems as “A System That Implements Sufficient Open 
Specifications for Interfaces, Services, and Supporting Formats to Enable 
Properly Engineered Components to be Utilized Across a Wide Range of 
Systems With Minimal Changes, to Interoperate With Other Components on 
Local and Remote Systems, and to Interact With Users in a Style That Facilitates 
Portability.” (Strei,2003, p. 6) The R2D2 ACB is chartered to ensure that the 
information that is collected for combat weapon systems such as the AEGIS 
platform allows for a reconstruction of events involving the weapon system. The 
reconstruction of events is not limited to test and evaluation but extends to 
events occurring during ship operations including ship deployments and the data 
recording and extraction capability to support the performance assessment of the 
combat weapon system. 

There are two parts to the OA Transformation: 

1. The OA Transformation Roadmap is designed to quickly rollout 
Navy Open Architecture (NOA). 
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2 . 


The Rapid Capability Insertion Process/Advanced Processor Build 
(RCIP/APB) to facilitate the insertion new capability within the Open 
Architectures as they are developed and become available. 


(Rushton, W. CAPT, USN et al, 2004, p. v) 
There are a number of platforms that are programmed for Open 
Architecture including the DD(X) Multi-Mission Destroyer and the CG/DDG 
AEGIS platforms as shown in Figure 6. With a large number of platforms 
converging to open architecture, the data recording and reduction requirements 
from one platform is likely to impact other platforms as these open systems are 
integrated together to meet the objectives of Network Centric Warfare through 
the implementation of FORCENET. While the introduction of COTS hardware 
has been implemented in some cases, OA will bring commercial software 
structures and designs into combat weapon systems to support the COTS 
hardware and complete the replacement of obsolescent MIL-STD systems. 



Figure 6. OA Transformation Roadmap 

(Rushton, W. CAPT, USN et al, 2004, p. 29) 
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The two programs that are specifically being researched in this study are: 


• AEGIS Open Architecture (AOA) 

• DD(X) 

Both of these platforms are Naval Surface Combatants that have been 
required to implement architecture that is compliant with Navy Open Architecture 
(NOA) standards. 


AWS COMPUTER ARCHITECTURE 
EVOLUTION PATH AFTER POM-04 



Figure 7. AEGIS Baseline Architecture 

(Williams, J., 2003, p. 4) 


The development of AWS has been incremental and evolutionary over 
time. As shown in Figure 7, early AEGIS baselines were MIL-STD equipment 
and COTS introduction began with Baseline 6 Phase 1. A demonstration test of 
the first all COTS AEGIS Baseline, 7 Phase 1 was completed in August 2003. 
AEGIS Baseline 7 Phase 1C, the first AEGIS baseline to implement AEGIS Open 
Architecture, is scheduled for demonstration testing in 2005. AEGIS Baseline 7 
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Phase 1C is accomplishing this through the use of evolutionary and spiral 
development. There are 3 spirals planned for AEGIS Baseline 7 Phase 1C. 
AOA is being implemented in 5 spirals as shown in Figure 8. The 
implementation of this architecture is spread across three areas that are: Radar 
Control, Weapons Control, and Display. 

Incremental Fielding of Capabilities 
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Figure 8. AEGIS Open Architecture Spirais 

(System Engineering Management Plan (SEMP) For AEGIS CG/DDG 
Open Architecture (Draft), 2003, p. 11) 


Through the use of evolutionary and spiral development, AEGIS will be 
able to build a little and then test the result. This strategy will be critical to the 
ultimate success of AOA since the adaptation of the system is occurring in 
spirals. The spiral development process presents challenges in the data 
dictionary, recording and reduction arena as well. As functionality is migrated to 
open architecture, the challenge is to ensure that the information required to 
evaluate the system is adequate and available. 
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D. DD(X) NEXT GENERATION DESTROYER 


The DD(X) multi-mission destroyer program is being developed to meet 
the NOA standards. The DD(X) platform is attempting to pursue advanced Open 
Architecture concepts including the Total Ship Computing Environment (TSCE) 
and modular design. As NOA has a Network Centric Warfare (NCW) 
implementation, the validation of data as it travels from one element to another 
element will be a critical component of the success of this weapon system. 
Figure 10 shows the transformational components in the DD(X) multi-mission 
destroyer which is designed for littoral and network centric warfare. 



Figure 9. DD(X) Multi-Mission Destroyer 

(DD(X) Multi-Mission Destroyer, 2004) 
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Figure 10. Breakout of the DD(X) Transformational Systems. 

(DD(X) Transformational Systems, 2004) 


The initial design includes the following transformational systems (DD(X) 
Transformational Systems, 2004, f1) that may require data recording and 
reduction: 


• Total Ship Computing System - The Local Area Network 
responsible for integrating warfare capability within the weapon 
system platform. TSCE will provide a Common Operating Picture 
(COP) by fusing together data provided by the onboard sensors 
and systems and data received from external sources. TSCE is 
designed to the requirements of NOA. 

• Advanced Gun System - Designed to provide greater firepower 
while being unmanned. Provides a land strike capability through 
the ability to fire advanced munitions and propelling charges. A 
GPS guided Long Range Land Attack Projectile (LRLAP) is one of 
the expected projectiles to be developed for this system. 

• Integrated Undersea Warfare - Designed to provide In-stride Mine 
Avoidance using a dual (HF/MF) frequency bow array 
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• Peripheral Vertical Launch System - Designed to launch 
STANDARD SM-2 and Evolved Sea Sparrow Missiles that are 
placed in a modularly designed launcher. By placing the launchers 
along the perimeter of the ship, reduces the loss of cells from a 
single hit. 

• Dual Band Radar - Integrates two active phased-array radars for 
detection of potential threats and fire control illumination during the 
engagement of selected threats. DBR will integrate the L-Band 
Volume Search Radar with the X-Band Multifunction Radar to 
perform search and track performance. 



Figure 11. Components of the Dual Band Radar. 

(Dual Band Radar, 2004) 


As the design of combat weapon systems such as the DD(X) becomes 
more automated in the pursuit of reduced manning levels, more information will 
need to be captured to ensure that system responses to internal and external 
stimuli can be reconstructed. As the first built-from-scratch open architecture 
combat weapon system, the data recording philosophy and structure will be 
carried into many future combat weapon systems as modules from this platform 
are reused in future systems. The development of the DD(X) platform as a 
network centric platform will provide opportunity to use open architecture 
components for other platforms on the OA Transformation Roadmap (Figure 6). 
Assuming a complete level of conformance to open architecture requirements, 
functional modules from one weapon system could be easily moved to another 
system. (Rushton, W. CART, USN et al, 2004, p. 29-32) 
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E. 


TRENDS IN DEFENSE 


As Secretary of Defense, The Honorable Donald Rumsfeld, stated on 4 
June 2004, “If the Department of Defense is to stay prepared for the security 
challenges of the 21st century, we must transform not just our defense 
strategies, not just our military capabilities, not just the way we deter and 
defend—but we must also transform the way we conduct our business.” (Glares, 
2003, p. 1) The R2D2 ACB must consider how this transformation will affect Test 
and Evaluation Events and areas within the R2D2 ACB charter. Since the R2D2 
ACB has been responsible for Naval Weapons Systems, the Navy’s vision 
outlined in SEA POWER 21 will be the guiding document for determining trends 
in defense that the R2D2 ACB should align to. 


SEA POWER 21 

Sea Shield 



Sea Basing 

Figure 12. SEA POWER 21 

(Clark, 2002, p. 1) 


In SEA POWER 21, the ability to share information is discussed. SEA 

POWER 21 states “Once information is acquired, it must be shared and 

processed to achieve knowledge dominance, leading to operational advantage. 

To meet this challenge, the Navy has been improving data sharing among 
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platforms, including Link 16, the Cooperative Engagement Capability...’’(Mayo & 
Nathman, 2003, p. 2). Link 16 and Cooperative Engagement Capability (CEC) 
are an integral part of the AWS and will be part of DD(X). As information is 
shared across many platforms, the data will become part of the system. NCW 
will allow the integration of the battlegroup to respond to threats more quickly. 
SEA POWER 21 “will require new models for command, control, and data flow.” 
(Mayo & Nathman, 2003, p. 1) resulting in the need to integrate data from many 
platforms to understand system performance. SEA POWER 21 will provide the 
warfighter with a complex set of data on which to base responses to possible 
threats. In this changing environment, “rapid information collection, analysis, 
dissemination, decision making, and execution are critical to winning the life-and- 
death race for combat effectiveness. Swift and effective use of information will 
be central to the success of SEA POWER 21." (Mayo & Nathman, 2003, p. 1) 

Part of SEA POWER 21 discusses the concept of Sea Basing that allows 
the Navy to be on the scene and in theater wherever needed. To meet this 
objective, the Navy is relying on new systems that are being designed to open 
architecture standards. These systems include the CVN(X) nuclear-powered 
aircraft carrier and the multi-mission DD(X) destroyer. The CVN(X) and DD(X) 
platforms will have significantly lower manning levels than previous generation 
aircraft carriers and destroyers. To reduce manning levels, more autonomy of 
the systems on the ship will be required, including the combat system. As there 
is less man-in-the-loop, validity of data in the system will become more critical to 
ensuring the quick and appropriate response to imminent threats. As advanced 
weapons and sensors are netted together, the requirements for data processing 
and transfer will continue to grow exponentially. 

Vice Admiral Albert Konetzni, the Deputy Commander and Chief of Staff 
for the Atlantic Fleet discussed the need for “solid intellectual analysis” (Costa, 
2003, p. 19) while transforming our defense capability. He discussed the lack of 
rigor in operation analysis in favor of stop light charts in PowerPoint 
presentations without substantive analysis to support it. (Costa, 2003, p. 19) The 

R2D2 ACB has provided a forum for supporting rigorous analysis of test events. 
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The R2D2 ACB has worked to ensure that the data required for analysis of the 
AEGIS weapon system is designed into the system. VADM Konetzni 
recommends “using realistic scenarios and experiments to make sure they will 
work as advertised”. (Costa, 2003, p. 20) By ensuring that the data required for 
analyzing these tests, the R2D2 ACB is working toward this goal. 

The importance of incorporating modeling and simulation to reduce the 
reliance on live fire testing is discussed in the Chief of Naval Operations (CNO) 
Guidance for 2004, Accelerating Our Advantages. The use of modeling is 
consistent with the reduction in live fire events that has been evident in the 
AEGIS program in recent years. The use of modeling is consistent with the Sea 
Enterprise initiative to “leverage technology to improve performance”. (CNO 
Guidance for 2004, Accelerating Our Advantages, 2003, p. 20) The AEGIS 
program has a history of using land based testing and simulated testing on the 
ship platform. The resolution of track identification in a joint environment will 
continue to present problems in assessing combat weapon system performance. 
The reduction of live fire test and evaluation events will require the collection and 
analysis of data from modeling and simulation combined with data from events at 
test ranges, land based test sites and, all other available test facilities in order to 
effectively evaluate system performance. 
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IV. RESEARCH ANALYSIS AND APPLICATION 


A. INTRODUCTION 


The design of the R2D2 ACB is intended to be similar to an Integrated 
Product Team that is limited to the areas affected by data recording. The ACB 
has a cross section of system designers and users. In this case, the users are 
the analysts who require data to perform their function. While IPT’s have 
authority to make decisions within the scope of their effort, the R2D2 ACB 
collects information and informs the program manager on data dictionary, 
recording and reduction issues. The ACB does not have the authority to 
implement a decision. The R2D2 ACB, unlike some IPT’s, does not choose its 
membership. Each activity assigns the people that they desire to support this 
effort and can remove or change members based upon changing priorities within 
the activity. This has resulted in the loss of some of the most productive 
members of the ACB; however, membership changes have provided the 
opportunity to find new champions for the R2D2 ACB within their respective 
organizations. 

The involvement of the R2D2 ACB has changed as recent AEGIS 
baselines have been introduced. In AEGIS Baseline 6 Phase 1, the R2D2 ACB 
did not exist until late in the design and development cycle. Accordingly, the 
ACB was able to assess past mistakes and try to determine lessons learned for 
future baseline development. The roadmap for AEGIS Baseline 6 Phase 3 
developments that were presented in the CNO Project 801 (DDG 51 Class) CNO 
Project 1669 (Cruiser Modernization) R2D2 In-Brief is illustrated in Figure 13. 
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Figure 13. Baseline 6 Phase 3 Development 

(Howard, 2004, p. 5) 


The R2D2 ACB was being established as the 6 Phase 3 Baseline was 
completing design and preparing to multi-element integration and test (MBIT). 
Baseline 6 Phase 3 provided an opportunity for the R2D2 ACB to effect system 
modification decisions through the Test Observation Report (TOR) mechanism. 
The TOR process provides a method for collection and prioritization of system 
performance anomalies detected during MBIT, engineering test and evaluation, 
system and demonstration testing of the computer program baseline. TORs are 
rated on a 1 to 5 scale based upon the risk to the program. The scale is as 
follows (ABGIS Performance Report for CBC OPBVAL, 2001): 
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Priority 

Description 

1 

Mission Faiiure or Safety probiem 

2 

Mission Degradation 

3 

1 or 2 with an acceptabie workaround. No workarounds 
aiiowed or safety probiems 

4 

Operator annoyance or nuisance 

5 

Not visibie to the operator or no tacticai impact (doc 
change). Visibie but triviai probiem, e.g. misspeiiing on 
aCRO 


Table 2. TOR Priorities 


(AEGIS Performance Report for CEC OPEVAL, 2001) 

Data recording issues were traditionally downgraded to priority 4 or 5 
since a ‘work around’ could be provided. This was appropriate in many cases. 
However, some data recording issues were found to affect the assessment of 
system performance during CSSQT and DT/OT testing. The ‘work around’ that 
was acceptable in a lab test environment was not available during the formal at- 
sea testing. In the lab, a test for a specific function could be conducted 
repeatedly, but in at-sea testing, the testing is very limited due to the high 
expenses of ranges, targets, ordnance, and test teams. A TOR Form is included 
in Appendix C. 

An example of a low priority observation occurred during an AEGIS 
CSSQT test event. A target was presented to an AEGIS destroyer for 
engagement to determine the effectiveness of the combat weapon system. 
When the target was ready to be engaged, the AEGIS Display System (ADS) 
displayed incorrect information to the console operator resulting in the operator 
choosing not to engage the target. The system displayed information from when 
the console had been in test mode rather than the present state of tactical mode. 
While the state of the console being in test mode or tactical mode was 
considered an operator annoyance (Priority 4) in the lab, the impact was much 
different during actual at-sea testing. 

AEGIS Baseline 7 Phase 1 provides the completion of the re-architecting 
of the AWS to COTS technology. The R2D2 ACB was established and able to 
influence the program development at an earlier stage than in Baseline 6. At the 
time of this writing, AEGIS Baseline 7 Phase 1 was preparing for its first CSSQT 
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testing with the USS PINCKNEY (DDG 91). A notional illustration of the 
scheduled testing is provided in Figure 14 as presented in the CNO Project 801 
(DDG 51 Class) CNO Project 1669 (Cruiser Modernization) R2D2 In-Brief. As a 
result of earlier involvement, a significant effort was undertaken to document 
extraction points and capture information that had been lost in the data 
dictionaries as a result of the introduction of COTS. While the results did not 
provide everything that the analysis community desired, significant progress was 
made. Early progress is critical for another reason. In the AEGIS baseline 
development, a capture of the previous baseline was performed for the starting 
point of the next baseline. Changes that are made to the computer program after 
baseline capture must be reinserted if it is to be included in future baselines. 
This has proven to be an expensive and difficult path to pursue. 



Figure 14. AEGIS Baseline 7 Phase 1 Deveiopment 

(Howard, 2004, p. 7) 


AEGIS Baseline 6 Phase 3 provides an excellent example of why early 
involvement is critical to getting an essential data recording capability into the 
baseline. The ADS displays tracks to the operator that are provided by the C&D 
computer via the ADS Track Server. During AEGIS testing, it was noted that 
some tracks did not appear at the console as expected. Review of the data 
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recording capability showed that the tracks could be found in the C&D Track 
Database file but there was no data to indicate when a track was actually 
displayed to the operator at an ADS console. The inability to determine whether 
a track was present at a console from the test data was determined to be an 
operational issue. Initially, the R2D2 ACB was unable to convince the system 
developer or program manager to implement a change in the Baseline 6 Phase 3 
computer program to collect and record this information. After a number of tests 
indicated the value of this information, the Program Manager became convinced 
of the importance of this data. The system developer was instructed to 
determine a method to collect this information and the associated cost. After an 
agreement was reached, the ADS Track File Data Recording (TFDR) was 
scheduled into the development path. Unfortunately, this change was not 
captured in Baseline 7 Phase 1. It remained an open issue as to whether the 
program manager for this baseline will capture this functionality. Flowever, even 
if this functionality is eventually captured into Baseline 7 Phase 1, it was too late 
for it to be captured directly into Baseline 7 Phase 1C and will likely be too late 
for Baseline 7 Phase 1R. 

As the development of Baseline 7 Phase 1C begins, the R2D2 ACB is 
attempting to be inserted earlier than in the Baseline 7 Phase 1 process. Figure 
15 illustrates the programmed development schedule for this baseline, which was 
presented in the CNO Project 801 (DDG 51 Class) CNO Project 1669 (Cruiser 
Modernization) R2D2 In-Brief. Baseline 7 Phase 1C is the first AEGIS Open 
Architecture Baseline and the ability for the R2D2 ACB to affect the development 
of the data recording remains to be seen. 
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Figure 15. Baseline 7 Phase 1C Development 

(Howard, 2004, p. 15) 

B. SUMMARY OF INTERVIEWS AND SURVEYS 


A survey was distributed to determine the effectiveness of the R2D2 ACB 
from the perspective of the core participants. The selection criterion was people 
who regularly attend and participate in the ACB. A total of eighteen people met 
the attendance and participation criteria and were surveyed. While there have 
been more participants in the history of the R2D2 ACB, the survey population 
represents the active core that has allowed the R2D2 ACB to perform its mission. 
Based upon discussions with Mr. Uy, R2D2 ACB Co-Chairman, the survey 
population was selected based upon the following criteria: number of meetings 
attended, knowledge of data dictionaries and data reduction, area of technical 
expertise and level of participation in meetings. The survey results are listed in 
Table 3. 
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Excellent or 

Extremely 

Effective 

Good or 

Very 

Effective 

Fairor 

Effective 

Poorer 

Ineffective 

No 

Opinion 

How effective in providing value to the AEGIS 
program has the R2D2 ACB been? 

33 . 3 % 

55 . 6 % 

11 . 1 % 

0 . 0 % 

0 . 0 % 

How effective do you expect the R2D2 ACB 
to be in AEGIS Open Architecture 
baselines? 

22 . 2 % 

55 . 6 % 

22 . 2 % 

0 . 0 % 

0 . 0 % 

How would you characterize the amount of 
time you are allowed to support R2D2 
issues? 

11 . 1 % 

33 . 3 % 

44 . 4 % 

11 . 1 % 

0 . 0 % 

At your activity, how well does the 
management/leadership understand and 
support your involvement in the R2D2 ACB? 

55 . 6 % 

0 . 0 % 

44 . 4 % 

0 . 0 % 

0 . 0 % 

How much benefit do you think the R2D2 
ACB concept would provide to a new 
weapon system program? 

77 . 8 % 

11 . 1 % 

11 . 1 % 

0 . 0 % 

0 . 0 % 


Table 3. Survey Results 

Each question will be addressed along with an analysis of the results and 
discussion of survey comments provided by respondents that are relevant to that 
question. One result that applied to the survey as a whole was that no answers 
of ‘No Opinion’ were received. As the criteria included participation, the level of 
involvement required to receive the survey provided a survey pool that had 
generally produces strong opinions on each of the subject areas addressed by 
the survey questions. Since the survey population consisted of active 
participants in the R2D2 ACB, this result was expected. 


1. Survey Question: How effective in providing value to the AEGIS 
program has the R2D2 ACB been? 

The first question intended to capture the perceived value that the R2D2 
ACB has provided. Most respondents (89%) believe that the R2D2 ACB has 
been very or extremely effective. This result is validated by the fact that the 
R2D2 ACB continues to exist. If the participants did not believe that the R2D2 
ACB was effective, they would elect to not participate and find other activities to 
engage in. Some program managers and system developers do not view data 
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recording as a critical component of the system under development. The 
experience of the R2D2 ACB has shown that data recording problems have been 
downgraded in favor of more visible tactical system software issues. (Johnson, 
2001) The R2D2 ACB is a champion of the importance of data recording to 
understand what needs to be fixed. By designing a robust data recording 
capability, system problems can be isolated more rapidly with less testing and 
solutions can be quickly verified. The introduction of COTS has allowed methods 
for debugging computer program development through methods other than data 
recording. Prior to COTS, the program developers depended upon the ATES 
data recording to obtain memory dumps that were used to determine program 
faults and errors. On the positive side, the opportunity to bring together the 
program developers, program managers and analysis community was viewed as 
beneficial in reducing redundant efforts and allowing the validation of problems 
and their importance. The R2D2 ACB provided the ability for open dialog in 
discussing the issues, mitigation, and planning implementation. With the diverse 
AEGIS community, the ACB allows understanding of the impact a change might 
have to another organization in near real time. As long as the AEGIS program 
continues to change, there will be a need to test it and work though issues that 
will arise - that is where the strength of the R2D2 ACB exists. 

2. Survey Question: How effective do you expect the R2D2 ACB to be 

in AEGIS Open Architecture baseiines? 

The second question is a forward-looking question to provide insight into 

the R2D2 ACB participant’s opinion of the future for the ACB. All future 

development for the AEGIS program is programmed for the AEGIS Open 

Architecture baselines once development of Baseline 7 Phase 1 is completed. In 

the opinion of 78% of the survey respondents. Program Manager buy-in is 

viewed as critical to the success in future baseline development. In the AEGIS 

OA development, the use of Technical Performance Measures (TPM) has been 

contractually employed. The use of TPM’s can be used to require data recording 

to be satisfied providing a motivation to the developer to ensure that specific data 
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recording capabilities are designed into the system. The R2D2 ACB could 
provide the Program Manager with the most critical areas to be measured and 
the requirements for a robust data recording capability. The R2D2 ACB would 
enable the data analysis community to become proactive in deciding what needs 
to be measured as the baseline is tested. Without a proactive approach, the 
data recording is likely to be an early casualty of budget and resource allocations 
and, as has happened in previous baselines, data recording will always be 
playing catch-up. Finally, the R2D2 ACB is in a unique position to see the data 
problems (both engineering and political) facing such a transition. 

3. Survey Question: How would you characterize the amount of time 
you are allowed to support R2D2 issues? 

The third question is intended to provide the participant’s perspective on 
how much effort they are able to provide toward the R2D2 ACB efforts. The 
survey results indicated that over 50% of the respondents characterize their 
ability the support the R2D2 ACB as fair or poor. Based upon discussions during 
previous R2D2 ACB meetings, two issues that have limited the ability of ACB 
members to support this effort are funding and collateral duties. The R2D2 ACB 
efforts are not separately identified in the funding documents. While the program 
office expects each activity to use baseline development funding for this effort, 
there is no specific funding allocated directly for this effort. When a conflict in 
responsibilities occurs, the bias is toward the tasks that are directly referenced in 
funding documents or statements of work. Other times, collateral duties can 
overwhelm the work schedule and result in R2D2 ACB efforts being minimized. 

4. Survey Question: At your activity, how well does the 

management/leadership understand and support your involvement in 
the R2D2 ACB? 

The fourth question is designed to correlate the previous question. In 
cases where an insufficient time is allowed, to what degree does this contribute 
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to management being unaware of the R2D2 ACB’s mission and benefits? For 
those who are satisfied with the amount of time available for R2D2 efforts, does it 
result in management support? Even in cases where the management 
understands the importance of participation in the R2D2 ACB, priorities can 
overtake R2D2 efforts when direct work traceable to tasking is at stake. Another 
problem is that some people receive good support while co-workers receive 
somewhat less support due to different managers perception of the R2D2 ACB. 

5. Survey Question: How much benefit do you think the R2D2 ACB 

concept wouid provide to a new weapon system program? 

The final question is designed to determine if the respondent believes that 
the R2D2 ACB is unique to AEGIS or if it can grow beyond this program. The 
critical concern is that the program managers understand and champion this 
effort. Every mission is going to require a data recording capability to measure 
its results. This is not unique to the Navy or DOD. NASA relies on telemetry 
data to determine if its space probes are operating correctly and commercial 
aircraft contain “black boxes” that contain data recording deemed critical to 
reconstruct aircraft and aircrew performance. A significant benefit that the R2D2 
ACB can provide is combining lessons learned and guidance on data recording, 
extraction, processing and analysis techniques and methods to programs in the 
process of defining and developing these capabilities. 

The survey results indicate that 78% of the respondents reported that the 
R2D2 ACB efforts would be extremely effective for a new weapons program. 
Since the ACB has been established for over 3 years, the positive views 
presented in the survey results are not the result of unguarded optimism for a 
new effort. Rather, it represents a time-tested reflection upon the 
accomplishments of this forum. By providing a forum where issues and 
deficiencies can be discussed without attribution, many R2D2 ACB members 
have been able to raise issues for investigation that would be difficult to carry 
forward at their activity. The R2D2 ACB can take action and allow the 


46 



membership to investigate issues as an ACB action rather than an individual 
concern. Furthermore, the management at an activity recognizes that action 
items assigned through the R2D2 ACB have been through a peer review process 
to determine that the issue is valid and important to the AEGIS program. 

C. R2D2 ACB AND RISK MANAGEMENT 


Based upon the experience of the R2D2 ACB to date, the primary 
functions that it performs are to review and monitor data dictionary and recording 
issues as the analysis community identifies the issues. The R2D2 ACB 
effectively performs risk analysis for the Program Manager. Each issue is 
identified and analyzed. Part of this process includes determining the impact on 
the program performance and schedule. As the R2D2 ACB evaluates issues, 
consideration is given to assess what information is lost resulting from each 
specific data dictionary and recording issue. Next, the risk resulting from each 
issue is documented through ACB meeting discussions and action item 
assignments. Once the issue is defined and understood, the analysis community 
provides feedback to determine how the problem may be addressed. Once an 
issue is defined and a solution identified, the Program Manager is provided the 
information necessary to determine whether the resources required that would 
resolve the problem is worth the tradeoff of resources for other uses. 

The R2D2 ACB provides consistency in prioritizing and investigation of 
data dictionary and data recording issues. Past experience in the AEGIS 
program has demonstrated a dependency upon the Program Manager. When a 
Program Manager is concerned about data dictionary and data recording issues, 
a significant effort is expended to resolve these issues. If a program manager’s 
decisions demonstrate that data recording is a low priority, other issues will 
absorb resources needed for fundamental data recording capability. However, if 
the R2D2 ACB is institutionalized, as it has been in the AEGIS program, the 
program manager has the technical resources readily identified and a chairman 

to describe the status of the data recording capabilities in the AEGIS baseline 
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development. Additionally, the institutionalization of the R2D2 ACB has served 
to educate members of the program office beyond the program manager, and 
develop the expectation that a fundamental data recording capability is an 
essential part of the AEGIS baseline development process because the value 
added has been shown in previous AEGIS baselines. Finally, the R2D2 ACB 
provides the program manager a motivated group of technical experts ready to 
help resolve problems as they are discovered. Through the use of technologies 
such as video teleconferencing, the R2D2 ACB has remained flexible and 
responsive in support of emergent data recording issues requiring investigation. 


D. APPLICATION OF STUDY 

1. Open Architecture 

The AEGIS program has been in the process of adopting COTS 
technology as quickly as possible without assuming too much risk in the 
program. Beginning in Baseline 6 Phase 1 and continuing into Baseline 7 Phase 
1, COTS technology has been inserted. Baseline 7 Phase 1 is the first AEGIS 
Baseline to have all COTS hardware for the AWS. The baseline development 
succeeding Baseline 7 Phase 1 are designed to be AEGIS Open Architecture 
baselines with the intent of capturing the complete benefit of COTS. 

The challenge facing the R2D2 ACB is how to respond to the changes 
presented by open systems. The introduction of COTS in Baseline 6 Phase 1 
required the development of the DXR data-recording format. The DXR format 
lacks features possessed by the ATES format that are useful to combat weapon 
system analysts due to the significant differences between the ATES and DXR 
formats. Each Spiral represents a new opportunity to provide technical support 
to the AWS baseline design and development team. Information on how to 
improve the data dictionaries and data recording from the previous Spiral can be 
provided. The R2D2 ACB will provide a resource to the design and development 
team as well for assessing the impact of changes to the AWS computer program 
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as they are considered rather than implementing and requiring rework to restore 
lost and required capabilities. 

The issues that the R2D2 ACB must resolve are critical to the successful 
evaluation of the combat weapon system performance and require a high degree 
of collaboration throughout the entire AWS analysis community. In order for the 
decisions of the R2D2 ACB to be respected and conformed to, the Program 
Manager must provide a clear and unambiguous level of support. For AEGIS 
Open Architecture, the real challenge is to continue to add value to the AEGIS 
program as it is converted into an open system. 

2. DD(X) Next Generation Destroyer 

The DD(X) platform is being designed to open architecture standards. As 
a new program, most of the people involved in the DD(X) program are unaware 
of the R2D2 ACB. The first step toward application of the R2D2 concept is 
information and education. The program managers in the DD(X) program are 
unable to implement a structure similar to the R2D2 ACB if they are not aware of 
its benefits. Additionally, the program managers need to be introduced to the 
people who can provide the expertise to initiate this type of ACB. Once the 
program managers decide to implement a R2D2 ACB, the system designers 
need to be introduced to the concept of the R2D2 ACB and the systems 
engineers who will be analyzing the performance of the combat weapon system 
need to be identified. With the program managers, system designers and 
analysts identified, the R2D2 ACB can start exchanging information. 
Traditionally, the R2D2 ACB has used the VTC format along with an electronic 
document management system. The choice of mediums used for information 
exchanges can be tailored to those that will be the most effective for the 
participants. Direct meetings can be more effective if all of the participants are 
closely located and email can facilitate a very large distribution list of participants. 
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Figure 16. DD(X) R2D2 ACB Structure 


With the development of a number of new systems, a possible application 
of the R2D2 ACB is tailored approach of semi-autonomous groups that are 
assigned to each element or group of elements. A possible structure for this 
approach is shown in Figure 16. By using functional working groups, the 
program managers could direct resources to the area or areas with the most 
critical problems that should reduce the overall risk in the area of data 
dictionaries and data recording. As the program progresses through its 
development cycle, the R2D2 ACB will be in place to support the formal Test and 
Evaluation events such as Developmental and Operational Testing that have 
relied heavily on recorded test data for system evaluation. 
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V. CONCLUSIONS 


A. INTRODUCTION 


The R2D2 ACB has been an effective tool available to AEGIS program 
managers since its inception in July 2001. Initially, the focus of the R2D2 ACB 
was on issues related to the ADS element of the AWS. As the R2D2 ACB 
continued to develop, issues related to each of the AEGIS core elements were 
investigated and potential solutions were proposed. While the team desires to 
see every solution implemented, the Program Manager is the final authority in 
determining which solutions are to be pursued. The Program Manager must find 
the funding for issue resolution and is in the best position to evaluate trade-offs 
between options. Based upon past experience of the R2D2 ACB, the Program 
Manager is not deciding between which data dictionary and data recording 
issues that should be funded. Rather, the decision is between data dictionary 
and data recording issues versus issues in other areas such as software code 
fixes or documentation. While the R2D2 ACB has had some significant 
accomplishments, change will be required if it is to continue growing into the 
future and increase the value that it can add to programs. 

B. KEY POINTS AND RECOMMENDATIONS 


Overall, the R2D2 ACB has been successful in its efforts supporting the 
AEGIS program. Participation has included most of the stakeholders in the 
AEGIS community who receive recorded data and rely on that data to perform 
systems engineering analysis. The key recommendations are: 

1. R2D2 ACB lacks direct funding. The R2D2 ACB lacks authority 
through funding mechanisms to prioritize problems in data dictionaries and data 
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recording. Until data dictionary and data recording is specifically written into the 
acquisition contracts and direct funding is attached, it will remain vulnerable and 
expendable. 

2. Membership stability. When data recording is not viewed as a 
high priority, the participation is inconsistent. Personnel turnover is inevitable in 
any program but the transition often does not encompass R2D2 responsibilities. 
The result is a loss of participation from some activities. 

3. Issues found during or after formal acceptance testing. 

CSSQT testing is a critical event in the delivery of the AWS for introduction to the 
fleet or preparation for DT and/or OT. It has been common to have many 
unknowns in the data recording capabilities entering these tests. During the USS 
Winston S. Churchill (DDG-81) CSSQT, the analysis team noted 24 weapon 
system performance analysis issues of which 7 issues were related to data 
dictionaries and data recording. With uncertainties in the capability of the data 
dictionaries and data recording, necessary data could be lost or not available 
resulting in the inability to characterize the system performed. A critical part of 
the CSSQT testing is to allow Test Qbjectives to be answered to characterize 
critical functionality with the AWS. This would hinder the productivity of the 
CSSQT and possibly leave many Test Qbjectives unanswered simply because of 
data recording deficiencies that may have been easily remedied prior to the 
testing. 

4. Collaborative Analysis. The R2D2 ACB allows the analysis 
community to bring data dictionary and data recording issues to the table that 
may have otherwise gone undiscovered during formal acceptance testing such 
as system demonstration tests. Working together, as the R2D2 ACB identifies 
an issue, the analysis community can understand and find solutions to help 
mitigate or resolve the issue. The collaboration has proven to be more effective 
than having each activity analyze issues independently. With the complexity of 
military systems, a collaborative approach will be essential in the future as it is 
unlikely that a single person or group will be able to evaluate system 
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performance. The R2D2 ACB provides the opportunity to bring together a 
focused group of experts capable of determining the data dictionary and data 
recording requirements to support the entire analysis community. Loss of the 
R2D2 ACB would likely result in a return to inconsistent analysis methods across 
system elements and activities. It is the membership that has made the R2D2 
ACB strong and viable. 

5. Closed Loop Systems Engineering Process. The R2D2 ACB 
has been a part of the closed loop systems engineering process that the AEGIS 
program has implemented. While the AEGIS program has only been able to 
achieve a partial implementation of the closed loop systems engineering process, 
R2D2 ACB can serve as the catalyst in continuing the closed loop systems 
engineering process into other combat weapon system platforms as the AEGIS 
program completes the baseline development efforts. 

By introducing the R2D2 ACB into the program at the initial design stages, 
the potential for significant reductions in data dictionary and data recording 
rework efforts exists. Data recording can help to produce a higher quality 
software product by assisting the development effort. A robust and effective data 
recording capability allows software rework to be objectively assessed and 
assists the developer in determining if software rework has caused inadvertent 
changes in the computer program performance. The experience of the R2D2 
ACB in the AEGIS program has demonstrated that a significant number of 
problems continue to exist beyond the rework to the AEGIS computer program 
that has already been done. 

6. Early Introduction in System Development. 

From the survey results, the general consensus is that the R2D2 ACB can 
provide great benefit and be very effective for a new weapon system program. 
However, it is critical to be involved early when the system is designed to ensure 
that the requirements in the system design include the infrastructure necessary 
to capture the data necessary to characterize system performance. Data 
recording is actually an important part of ensuring that a system will be 
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supportable. The information on system performance that is provided by data 
recording and reduction can provide evidence of system deficiencies and 
reliability, maintenance and availability. In fact, as testing for events such as 
early operational assessments are integrated into program schedules, the 
effectiveness of the R2D2 ACB will increase. Early involvement applies to 
existing systems that are being upgraded as well. As the next generation of data 
recording is created within the AEGIS Open Architecture design, the R2D2 ACB 
will need to be involved in the definition and implementation of the data dictionary 
and data recording formats as they are designed. 


C. POTENTIAL AREAS FOR FUTURE RESEARCH 


Many areas exist for future research on this topic. Since this study has 
been limited to combat weapon systems within the Navy, future research could 
pursue the applicability of the R2D2 ACB to weapon systems in other branches 
of the military service. Additionally, programs that rely on mission critical 
systems such as the Air Traffic Control Systems of the Federal Aviation 
Administration (FAA) and the Space Shuttle Program of the National Aeronautics 
and Space Administration (NASA) are candidates for future research as well. 


D. CLOSING REMARKS 


The results of the survey indicate that the R2D2 ACB can provide value to 
a weapon system program but value is not the only criteria that a Program 
Manager must consider. Two additional considerations are whether the R2D2 
ACB aligns with the Defense Transformation efforts underway and will it reduce 
risk to the program in a concurrent development environment. For the Defense 
Transformation efforts, this analysis will focus on the transformation as provided 
in SEA POWER 21. As the Navy moves toward Network Centric Warfare, the 
data shared and transferred among systems will become more critical than in 
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previous generations of combat weapon systems. Another consideration is in the 
area of concurrent system development. Concurrent system development 
results in early operational assessment of system performance. As formal 
system testing is conducted earlier in the system development, the capability to 
assess system performance is being required earlier in the system development. 
Data recording and reduction provide the objective evidence to establish system 
design maturity and document operational issues in system performance. The 
R2D2 ACB has demonstrated that it will provide value to the Program Manager 
by prioritizing data recording and reduction capability requirements allowing the 
Program Manager to trade-off value added among competing priorities. 


55 



THIS PAGE INTENTIONALLY LEFT BLANK 


56 



APPENDIX A. 


A. TOR FORM 


To: LMSDSSAAWCS 


c/o Computer Sciences Corporation 
Aegis Configuration Management Office 

P.O. Box N, Moorestown, NJ 08057 


CONFIGURATION MANAGEMENT INFO. Page 1 


1. TOR Log 

2. TOR Number 

Date 



T 


ORIGINATOR INFORMATION (Required) 


3. Originator Name 

4. Organization □ RAYTHEON 

□ LMSysEng □ CSC □ TECHREP 

□ LMCPP □ NSWC □ ATT 

□ LMS/WDev. □ CMU 

5. System Element 

6. Date Observed 

7. Return Address 

8. Program 

Identification/Loadfile 

9. Baseline 

7P1 

10. 

11. 

12. Test Site (where observed) □ NSWC □ ACSC (Wallops island) 

□ Desk Check □ PGCD CPTSQ CSEDS □ PTC 

□ □ 

13. Test type □ MEIT □ System Level Eng. Tests (ISEs, Stress, Endurance, etc.) □ SQT/EQT 

□ Shipyard Checkout □ Element Int. □ CSIT □ PTC Checkout/Production Acceptance Test □ Shipboard 

Operation 

□ Dev/SE Checkout □ ET&E □ Formal Demos, DTs/OTs,etc. 


ORIGINATOR INFORMATION (As applicable) 


14. Test 

Environment 

15. Test Procedure Used 

S S 

cript top 

16. Dump/Recorded Data 

TDD 
ape Nos.: ump ata 

D P 

17. Documentation Affected 

18. Module/Function Affected 

19. Site Log No. 

19a. ITT Cross Reference 

19b. Japan CPCR Number 

20. Associated TDR 


ORIGINATOR OBSERVATION (Required) 




21 

Sh 

ortT 

itie ( 

Refit 

3CtiV 

3 of 

Obse 

3rvat 

on) 





















22. Description of Observation, GMT (if applicabie), and other comments (Unciassified Information oniy) 


_ Attachment □ 

23. Effect on System Eiement when observation described occurs and additionai System Engineering comments (if 
appiicabie). (If recommended operational priority is 3, describe workaround.) 


Pri 

Check one of each or N/A: 



Sec. 

Recommended 1679A Operational 

□ 1 □ 

2 □ 3 □ 4 □ 5 


Recommended Operational 

□ AD 

LU 

□ 

Q 

□ 

o 

□ 

m 


24. Locai Probiem Report (LPR) Determination 
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a. QALoad/CTL? DYES □ NO If yes, do not 

b. If no: 

Problem in new development code? D YES D NO D 

c. If b is yes, identify # of CRCR that 
and treat TOR as an LPR 


25. 

CPCR# 


(For 

non-LPRs 

SIB Pri 



CPCR Sub 
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APPENDIX B 


A. EXAMPLE PRESENTATION FOR R2D2 ACB 



BASELINE 6 PHASE 3 
DATA ANALYSIS 
LESSONS LEARNED 



PA$11 6f29f2004 
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Preparing for CSSQT 

• CSEDS Testing 

• Analysis Tools 

• Dahlgen Test Team Support 

• Shipyard and beyond 

CSSQT Lessons Learned 

• Event Example 

• Data Quantities 

• Tools 

• Timelines 

Have we learned our lessons? 


PfiS12 WJWOW 
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Preparing for CSSQT 


CSEDS Testing 

■ In Baseline 6 Phase 3, it became apparent very early on that the 
status quo was not going to work. Experience with Adjunct data from 
6 phase 1 drove us to CSEDS. 

■ Functions at CSEDS: Tool development and debugging, system 
knowledge, interfacing with design agent, DEMO testing support in 
DTD analysis and verification. DEMO testing is more than SPY 
analysis. 

■ Analysis experience with CSSQT Dry Runs! Testing of tools with data 
that is as comparable to real life as can be achieved in a lab setting. 

Without the knowledge and analysis process and tool 
development, the completion of event reconstruction 
of the 6 Phase 3 CSSQTs would not have been very successful 


PASIS WJWOW 
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Analysis Tools 

Analysis Tools required more effort than expected 

■ Configuration Management has been implemented 

• Challenges still continue in the trade-off between 
responsiveness versus control 

■ CSEDS testing provided some structured testing and in some 
instances the ability to redo some things. 

• Trying to redo things at CSSQT is expensive and often just can’t 
be done 

• Written procedure makes it much easier to verify analysis tool 
performance 

Lack of CEO integration during the CSEDS tests 

resulted in some unexpected tool deficiencies 

PfiS14 WJWW 
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Surface V/Birfarft C»ntw DNtsloii | 


CSSQT Example 
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[Stfj^W /sj Evaluation Criteria For Surface 

Surface VAirfare Centar DNlGtoii Missile System (SMS) Performance 


NAVSEA INSTRUCTION C8820.1 

• To provide criteria for assessing SMS combat systems 
performance during AAW non-combat firing operations. This 
criteria provides the measure for assessing performance against 
air and surface targets, consistent with realistically observed 
data. 

The Criteria 

• Combat system performance during a target presentation 
consists of three phases 

• Surveillance Phase 

• Target Engagement Phase 

• Missile firing Phase 
AWS Test Objectives 

• Primary - 226 

• Secondary - 58, 302, 441 


WJWOW 
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A23-D8602 


Surface v/arfara Centar DNistoii 


A23-R65d 


A23-D8602 


□ BQM34S, SSJ HP BlinkSjG, DSQ50 
□ALT:0.05Kft, SPD: M0.7 

□ Heading: 1475 T 
□30 DM IP, 0 DM CPA 

□ NKC-136 SOJ LP Barrage 

□ Heading: 145 T 

□ 175 DM IP 


W-532 


DOC as Trach Pont: 
a60«TQ20DMirom DDGas 
SS« 20 ' NM 1 S* 4 a' W 




DDGae nre Pont 

iVW N 
115'45- W 

" v Vi ' 


Notto exact Scale 


W-289 


PA317 5f2W3C04 
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How It Will Be Scored 


One Presentation 
One Target 

Demonstrate Depth Of Fire 


Coverage Area 
TO-226 Primary 



NKC 


Coverage Area 
TO-441 Primary 

Complies With C8820.1 
Two Primary TO’s Can Work 


PA316 
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Surface V/Birfarft C»ntw DNtsloii | 



Normal operation is for the data 
processor on the ship to take the 
optical and divide it into its 
partitions, zip it and send it to 
Corona. 

For 15 Partitions it takes 45 min to 
transfer all compressed files. (3 
minutes per partitions) 

Once received, the on site data 
processors do a sort and spht of 
the partition into its “ATES” and 
“DXR” Components. For 15 
partitions this takes 30 minutes. 
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□Track Data 


uC&D Traffic 
■Tip Data 

□Search Data 



Optical, - 2gig 
Once received is 
split into up to 
30 time aligned 
partitions. 


PA3110 MSWCKW 



Data is NOT time aligned, but recorded 
as a function of when a buffer is full, or 
during a preprogrammed periodic, it 
will be sent to the optical 


ATES 


DXR 


For Partition 
7, all data 
can be 
extracted 
because of 
the luck in 
buffer 
sequence 
loading 


For Partition 15, to get all 
the data you have to go back 
to partition 12, and maybe 
into the next optical for the 
data 
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What it takes 

■ For every hour of testing in manned raids, it takes 4 hours 

• To get the data here, 

• Process the data, 

• Evaluate the test objectives, 

• Put together the report 

■ CEC data is a critical player in every event. Remote CEP data is 
the driving source of most engagements. Every live fire 
engagement for the recent 86/88 CSSQT had CEC data in it 

■ DXR data is cumbersome to go through. Measured in GB instead 
of MB! 

■ Limitations still exist 

• We have no clues as to v\/hat happening on the LAN 

• Adjunct switching and reconfiguration is not recorded 
Since DDG-85, data prep time required for event analysis 

has doubled and in SPY and C&D tripled. 

PAS111 MSWOW 
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Lessons Learned? 


■ We will be doing BASELINE-7 

■ All COTS. 

■ Optical’s are 5 GB, twice the capacity as 6 Ph III 

■ New Programs 

■ All DXR data 

■ All new Hardware with increased performance capability. The 
big Littoral radar system 

■ Anticipate more analysis time than with the 6 Ph III ships 

The key to success is early, sustained effort 
wherever the opportunities exist 
Teaming with TECHREP, NGIT, Dahlgren, LM 
and all other stakeholders 

PAS112 WSWOW 
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